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FREE SPACE MONITORING SYSTEM FOR MOTOR VEHICLES 



Backgroxuid o£ the Invention 



Field of Invention 



[0001] 



The invention concerns a process and device for 



monitoring the free space in the direction of travel of a 
vehicle according to the precharacterizing portion of Patent 
Claims 1 and 10. 

Related Art of the Invention 



the driver of the motor vehicle to predict the precise track of 
the vehicle and to know how much free space will be needed in 
order to avoid collision. This is the case in particular when 
the vehicle operator is not familiar with the dimensions or the 
maneuvering characteristics of the vehicle. 

[0003] In order to support the vehicle operator during 
difficult driving maneuvers, an operator information system has 
been developed as described in Japanese Published Application JP 
2000-67395 A, wherein image data describing the roadway have 
predicted vehicle tracks superimposed thereupon. In this manner 
it becomes possible for the vehicle operator to better estimate 
the future behavior of his vehicle. To do this the system uses 
the data of two sensors, which are mounted on the front wheels 
of the vehicle and therewith provide information regarding the 
direction of motion and the speed of the vehicle. From the 
sensor data a future vehicle track is predicted and this is 
superimposed as a line structure of a depiction of the vehicle 
track in a display unit. A disadvantage with such a system is 
that the predicted vehicle track is superimposed only upon a 



[0002] 



In complex vehicle maneuvers it is often difficult for 
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display of the vehicle roadway or route, with no images embedded 
in the display providing the operator information regarding 
objects in the vicinity of the vehicle. 

[0004] In order to provide the vehicle operator an improved 
overview over the actual environment of the vehicle and to allow 
better maneuvering therein, an improved, environment monitoring 
system is proposed in Patent Application EP 1 158 803 A2 . 
Herein the situation of the environment of the vehicle is 
detected using multiple camera systems, with data transformed 
into a birdseye view. Into this image a vehicle system 
representing the vehicle is embedded in a position corresponding 
to the vehicle position. The vehicle symbol is herein so 
designed, that it fits into the environment image with its 
correct size relationship. The image information provided to 
the vehicle operator thus corresponds to the view from the point 
above the vehicle looking down onto the environment; that is, 
the vehicle and its environment are represented in a type of 
birdseye perspective. In order to support the vehicle operator 
in his evaluation of the future travel path of the vehicle, the 
image has superimposed thereon a three dimensional line contour 
describing the predicted vehicle path of travel. The line 
contour represents the image of an edge or frame of a rectangle, 
which is curved along the predicted path of travel. The two 
lower longitudinal edges of the rectangle lie on the vehicle 
path of travel and correspond substantially to the vehicle track 
represnted in JP 2000-67395 A. The height and breadth of the 
rectangle is herein selected to be smaller than height and 
breadth of the vehicle. The vehicle operator obtains by this 
mode of representation on the one hand a good depiction of the 
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future relationship of the vehicle to its environment; however, 
due to the perspective of representation which is unconventional 
for the vehicle operator it is not possible for him, in simple 
manner to grasp the danger of collision, which would be 
attributable to the parts projecting from the vehicle sides into 
the area of the predicted vehicle path of travel. In 
particular, the parts projecting into the predicted vehicle path 
from above can only be represented and recognized with 
difficulty. It is also not possible to represent for the 
vehicle operator thoroughfares (tunnels or entryways, garages) 
in a plausible or intuitively iinderstandable manner. 

SUMMARY OF THE INVENTION 
[0005] It is thus the task of the invention to provide a 
process and a device which makes possible for the operator of a 
vehicle to envision the future vehicle path of travel with 
regard to its environment, and to reliably recognize dangers of 
collision due to parts projecting from the side or upwards into 
the predicted vehicle path of travel. 

[0006] This task is solved by a process and a device with the 
characteristics of Patent Claims 1 and 10. 

[0007] Advantageous embodiments and further developments of 
the invention can be seen from the dependent claims . 

[0008] In a process for monitoring the environment space in 
the direction of travel of the own vehicle, image data of the 
environment around the vehicle in the direction of travel of the 
vehicle is recorded using a camera system. Further, the three 
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dimensional free space required for unimpeded travel is 
calculated in advance in a signal processing unit on the basis 
of the operating parameters and the dimensions of the vehicle • 
The operator of the motor vehicle thus has displayed to him in a 
display at least those parts of the image data obtained from the 
camera system and corresponding to the area of the required free 
space. In inventive manner, the image data are herein displayed 
substantially from a perspective which would represent the 
environment in the path of travel of the vehicle from the direct 
perspective of the operator of the vehicle. Further, in 
inventive manner the image data associated with the required 
free space is subjected to a further processing, whereby as a 
result of this further processing, the vehicle operator is 
informed regarding whether or not a sufficient free space is 
available to him for unimpeded travel. It is thus possible 
therewith, in inventive manner, by a continuous evaluation of 
the image data, to automatically react to a dynamic change in 
the vehicle environment and to inform the vehicle operator as to 
whether a sufficient free space is still available or not for 
unimpeded travel. By the display of at least the part of the 
image data representing the required free space, the vehicle 
operator has the possibility of independently assessing and 
evaluating the potential of danger in the predicted path of 
travel. The display of the image data occurs therein not in the 
perspective of the representation according to the systems known 
in the state of the art, in which information is presented as 
viewed from above (birdseye view) , but rather essentially from 
the perspective wherein the environment around the vehicle would 
appear to the vehicle operator as it would in his own direct 
view in the direction of travel. It is herein however also 
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conceivable, that the depiction of the vehicle environment, in 
particular as dictated by the optics of the camera system (for 
example, wide-angle lens or catadiopter camera lens) , is 
displayed distorted. On the other hand it is however also very 
conceivable, that in the framework of the further data 
processing any warping or distortion of the image data is 
corrected. In a manner which is understandable to him, the 
vehicle operator sees an image depicting the environment, of 
which the perspective substantially corresponds with that of his 
own view, as he would himself perceive by directly looking out 
of the window. It is thus no longer necessary that the vehicle 
operator take the unaccustomed birdseye view perspective 
representation and to convert it into the real conditions. 

[0009] Data regarding the dimensions of the vehicle are 
stored in a memory, and is accessible to the signal processing 
unit, in order, with the aid of the operating parameters of the 
vehicle, to be able to estimate the necessary driving space for 
the driving maneuver. Therein there is the possibility that the 
dimensions stored in the memory do not correspond with the 
actual dimensions of the vehicle, but rather are expanded by a 
certain safety margin. One such safety margin could in 
advantageous manner be up to 20 centimeters. It is also 
conceivable that this safety margin is manipulated by the signal 
processing unit to be variable and, depending upon the vehicle 
operating speed and/or danger potential, that it becomes 
enlarged or reduced. It is also possible that the safety 
margins are variously adjusted for the various areas or parts of 
the vehicle. Thus it is for example conceivable that the safety 
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distance from bumpers is set smaller than that for the contact- 
sensitive rear-view mirrors. 

[00010] In inventive manner, this memory for recording the 
vehicle dimensions is so arranged and organized, that therein 
there are stored a multiplicity of data sets having different 
vehicle dimensions, which can be individually selected by a 
targeted selection, and provided to the signal processing. The 
memory can also be provided with a means, by which at least one 
of the data sets stored in the memory can be changed out. This 
provides the advantage, that the system for monitoring 
environmental free space is relatively easy to adapt to changes 
(for example in the vehicle contour, e.g. new trailer, different 
vehicle superstructure, etc.). Precisely such a design is of 
great advantage above all in the case of operation of a fleet of 
vehicles, in particular trucks with variable superstructures. 
In this way even electro-technologically unsophisticated 
operating personnel can operate the inventive system using 
simple input elements. They can do this without change of 
electro-technological components or connecting with or operating 
programming devices. 

[00011] In particularly preferred manner these changes in the 
dimensions of the vehicle can also be assumed or incorporated in 
simple manner in the memory associated with the signal 
processing device, in that the data set describing the new 
dimensions are read into the memory in simple manner via a 
mobile or stationary data transmission unit using a suitable 
interface. In particularly advantageous manner, in the case of 
a suitable design of the system the inputting and the selection 
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of data sets can also occur wirelessly, via a communication 
device. In the case of frequent repetitive (identical) changes 
(for example driving with loaded or as the case may be unloaded 
container) it also possible that multiple various vehicle 
contours or dimensions are stored in memory of the data 
processing unit, and can be called up at will. In particular, a 
keypad accessible by the vehicle operator can be provided for 
this purpose. In particularly simple and useful manner, this 
keypad has individual keys associated with the data sets stored 
in memory, so that with a single key stroke selection the 
desired data set can be called up. 

Brief Description of the Drawings 

[00012] In the following the invention will be described in 
greater detail on the basis of the illustrated embodiment shown 
in the figures. 

Fig. 1 shows an image scene (10) in which the edge borders of 



the free space are visualized by superimposition of 
walls represented in symbolic form (16) . 



Fig. 2 



shows an image scene (20) identical to Fig. 1, wherein 
here the edge borders of the free space are visualized 
by superimposition of a winding symbolic represented 
virtual driving tube (26) with checkerboard pattern. 



Fig. 3 



shows the relationship of the global coordinate system 
to the vehicle coordinate system. 



Fig. 4 



shows the image geometry of a parabolic mirror. 
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Fig. 5 shows the relationship between the coordinates of an 
omnicam and those of a pinhole camera. 

Fig. 6 shows the inverse transformation of the image position 
of an omnicam with reference to a pinhole camera 
model . 

Fig. 7 shows the image recorded by an omnicam in the 
direction of travel, in which the horizon is 
illustrated. 

Fig. 8 shows the optical flow field of the image points 
within the scene. 

Fig. 9 shows the optical flow field for a wall, which rises 
perpendicular to the optical axis of the omnicam, in 
the case of movement along the optical axis. 

Fig. 10 shows the optical flow of a sidewall during the 
movement of the vehicle. 

Fig. 11 shows the optical flow of a horizontal plane of the 
movement of the vehicle. 

Fig. 12 shows the depth map of a corridor. 

Fig. 13 shows the optical flow for the corridor of Fig. 12. 
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Fig. 14 shows, color-encoded, the flow field for the 
horizontal flow shown in Fig. 13. 

Fig. 15 shows, color-encoded, the flow field for the vertical 
flow shown in Fig, 13. 

Detailed Description of the Invention 

[00013] After the path of travel and the therefore required 
free space has been predicted by the signal processing unit, it 
is particularly advantageous when in the framework of the 
further processing of the image data any dynamic time-based 
changes of this data are taken into consideration. In this 
manner the system can recognize objects or persons which enter 
into the free space required for the vehicle maneuver. If this 
is the case, then the operator of the vehicle should be informed 
in an appropriate manner that sufficient free is no longer 
available. This information can occur in particular by optical, 
acoustic or haptic perceivable s ignal -providers . On the other 
hand, it is also conceivable that the image data displayed to 
the vehicle operator of objects or persons representing an 
impediment in the free space would be represented visually in 
enhanced or accentuated form. For this, there is suitable for 
example a color enhancement or false color accentuation of the 
display, in particular a false-color representation, increasing 
in steps depending upon the danger of collision (for example: 
high collision danger -> object is colored red; slight danger of 
collision —> object is colored yellow) . In particularly 
preferred manner the objects or persons representing an 
impediment could also be highlighted in the manner that symbols 
are superimposed thereon. For example, in the framework of the 
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further processing of the image data, objects which have moved 
are detected by means of an image processing algorithm based 
upon the optical flow, then in suitable manner the moved object 
can have superimposed symbolic movement vectors (arrows) which 
describe the direction of movement and movement speed (length of 
the arrow) . 

[00014] A particularly advantageous mode and manner of 
informing the vehicle operator regarding the amount of free 
space available to him is comprised therein, that in the 
framework of the further processing of the image data, the image 
data displayed to the vehicle operator has superimposed a 
symbolic representation of the boundary limits of the required 
free space as predicted in the signal processing unit. Therein 
it is particularly advantageous to describe the edge boundaries 
by walls represented symbolically. The height of the walls 
corresponds therein to the height of the required free space 
(that is, essentially the dimensions stored in memory regarding 
the maximum height of the vehicle) . 

[00015] In particularly advantageous manner the edge 
boundaries are so represented that the required free space is 
bordered by a type of ''driving tube'' or ''driving tunnel"; for 
this, the edge boundaries occur by symbolic represented walls 
and the overhead area in the manner of a tube. By this "tube 
view" the attention of the vehicle operator is specifically 
directed, and it also becomes possible for him with greater ease 
to substantially recognize objects which project into the needed 
free space from above or from the sides. 
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[00016] For the driver, the estimation of whether spatially 
fixed objects extend into the rearward free space becomes 
particularly simple when, during the (backwards) travel, he can 
compare the relative movement of objects in the image (display) 
with the relative movement of a spatially fixed symbolic driving 
tube: if the objects move faster than the boundaries of the 
driving tube, then they are spatially closer - that is - within 
the direct area of danger. If during travel they move slower 
through the image, then they are outside of the driving tube and 
there is no danger that they will be contacted during further 
driving. For this reason it is advantageous to so design the 
walls and overhead, and in certain cases even the floor, of the 
symbolic shown tube, such that it 

1. ) has a geometrically clear and distinct identifiable 

structure (for example: grid pattern, checkerboard pattern 
or series of columns) 

2. ) provides a "through view" to the video scene lying there- 

behind (for example: broken away or semi-transparent 
structures of the walls and/or the ceiling) . 

3. ) is respectively so calculated or, as the case may be, 

determined based on the vehicle movement, that it remains 
stationary in the longitudinal direction. 

[00017] In order to make the superimposition of the camera 
image with the symbolic depiction of the edge boundaries of the 
required free space appear as realistic as possible, the image 
characteristics (transformation equations) of the camera image 
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are to be taken into consideration. For a better understanding, 
in particular with regard to the realization of the advantageous 
design introduced above under point 3 (standing still in the 
longitudinal direction) of the graphic overlay, in the following 
there will be discussed as an example the necessary 
transformation equations for the superposition of an image 
recorded by means of a catadiopteric camera (omnicam) with a 
parabolic mirror. Therein, the basic steps and, in detail, the 
individual steps, of the transformation equation are explained 
and elaborated, which are necessary for the inventive 
super imposition of the camera image with the edge boundaries of 
the required free space. 

[00018] Of course the invention is not limited to the 
employment of such a camera, but rather necessary 
transformations explained on the basis of this camera geometry 
in detail allow themselves to be directly applied in simple 
manner to other camera systems with other mapping or 
transformation characteristics. 

Coordinate Systems 

[00019] The coordinate systems used herein follow the 
conventions from the literature regarding optical flow. In the 
3D-camera coordinate system the Z-axis indicates the direction 
of view, the X-axis to the right and the Y-axis downwards. The 
center image coordinates are provided by an x-axis towards the 
right and a y-axis directed downwards. The origin of both 
coordinate systems is the center of the image. 
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Regarding Environment -to- Vehicle Coordinates 

[00020] For a camera mounted on a moving vehicle, one can 
first introduce two important coordinate systems: an environment 
coordinate system, in which the vehicle environment exists, and 
a vehicle coordinate system, which moves along with the vehicle, 
see Fig. 3. When the vehicle moves in the direction of the Zw- 
axis, the transformation between these two coordinate systems is 
described by the position W{t) of the vehicle in the environment 
coordinate system: 







(x ) 
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Yw 




0 
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[00021] The instantaneous speed of the vehicle is V = W. 
Therewith, the time derivation of the point in which the vehicle 
environment rests is: 









-- 0 







Regarding Veliicle-to-Camera Coordinates 

[00022] It is shown in Fig. 3 that the direction of view of 
the camera is tilted about an angle x against the (backwards) 
direction of movement. In this case the transformation of 
vehicle to camera coordinates occurs as follows: 
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[00023] In the general case, these transformations contain not 
only the tilt angle x, but rather the three Euler angles. 

[00024] From Equation 2 one obtains the movement of a 
stationary point in the camera coordinates 



9 

X 

# 

Y 



0 

z 



V sin T 
-V cos X 



(5) 
(6) 
(7) 



Polar Camera Coordinates 

[00025] The arctan (y,x) - function with two arguments 
determines the phase of the complex niimber x + iy in the value 
range [-7C, 7C] . The sequence of the arguments were selected 
following the standard or paradigm of the ANSI-C function atan2. 

[00026] Transformation of polar into Cartesian coordinates: 

X = R sin 9 cos <|) (8) 
Y = R sin e cos <|) (9) 

Z = R COS 0 (10) 
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[00027] Transformation of Cartesian into polar coordinates: 

R = Vx' + + (11) 

0 = arctan (V^^TF^z) (12) 
0 = arctan (Y,X) (13) 

Polar Image Coordinates 

[00028] Transformation of polar into Cartesian coordinates: 

X = r cos <|) (14) 
y = r sin <|) (15) 

[00029] Transformation of Cartesian into polar coordinates: 

r = yjx^ ^y^ (16) 
(|) = arctan (y,x) (17) 

Point o£ Intersection o£ Direction o£ View witli Plane of Object 
Space 

[00030] The vector of the direction of view w is provided in 
the vehicle coordinates. In the case ivi # 0, the beam cuts 
through or intersects the plane Xv = a at the point 



p = — w (18) 



[00031] In the case W2 0, the beam intersects the plane Yv 
Jb at the point 
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b 

P = — M/ (19) 
W2 



[00032] In the case w3 0, the beam intersects the plane Zy 
c at the point 



p = — w (20) 
W3 



Omnicam Transformation for Image or Napping or Picture 
Mirror Form 

[00033] The transformation through the omnicam modeled here 
occurs in two steps: a reflection in a parabolic mirror occurs 
from an orthographic image. 

[00034] The parabolic mirror surface with a focal length f is 
defined by 



^ = / -r- ' (21) 

4/ 



[00035] This expression is used in many of the following 
equations. That means, in those cases z is for the term on the 
right side of the equation, which depends from the position (x, 
y) . Since the focus of the parabola originated in coordinate 
data, the direction of view at the image point (x, y) is defined 
by 
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r 



V 



(22) 



[00036] The derivation of the parabolic equation (21) is then 



dx 



X 

2f 



(23) 



2/ 



(24) 



[00037] That means, the normal vector on the mirror surface 
has the direction 



m 



^x ^ 



y 



(25) 



[00038] Therein it is not postulated or claimed that s(x, y) 
or Jn(x, y) are unity vectors. 

Projection of Points 

[00039] The omnicam image is represented most simply in polar 
coordinates : 



r = 2/ tan — 
2 



(26) 
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I • 

[00040] As discussed above, t is the distance from the image 
center, measured in pixels. 9 is the angle of the image 
direction to the optical axis. This relationship can be derived 
with the aid of Fig, 4 and Equation (25) . The azimuth ^ is the 
same in the camera and image coordinates. 

[00041] In Cartesian coordinates this means 



2JX IjX 
X = — ^ — = ^ (27) 



y = — — = , (28) 



Projection of Lines 

[00042] A straight line in the space is defined by a point p 
and a direction / = (/i, I2, h)^ with | / | =1. The vector n = p 
X 1 is perpendicular to the plane, which is spanned by the 
straight line and origin of the camera coordinates. The camera 
components of n determine the type of the omnicam image of the 
3D-lines: 

• in the case 21 = 0, the line cuts through or intersects 
the origin and its image is a point with the 
coordinates 



X = ^ (29) 
l-h/3 
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* • 

y = (30, 



• in the case n ^ 0, however na = 0, then the 3D~line 

parallel to the optical axis and its image is a 2D- 

line through the image center with the equation 

n^x + n^j; = 0 (31) 



in the other case, the image of the line is a arch or 
circle segment with the center point (Xc-Yc) with a 
radius r: 



Xc = 2fni/n3 (32) 
Yc = 2fn2/n3 (33) 
r = 2f|nil/n3 (34) 



Distance from Plane 

[00043] The length of the vector s(x,y) from Equation 1 (22) 
is 

\six,y'\ = yjx' W = / + = 2/ - z (35) 
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[00044] Thus, a point with the distance R 
position (x,y) has the camera coordinates 



Y 



= R 



(36) 



[00045] This produces a compact relationship between the 
camera coordinates of the object point on the one hand as well 
as distance R an image position (x,y) on the other hand: 



R 



2f-z 



(37) 



or in greater detail 



(X^ 
Y 



r 



R 



+/ 



4^ 
4^ 



4/ 



2 _ ^2 _ 



(38) 



[00046] The distance of a perpendicular wall with the equation 
X = a is provided for an image point (x,y) by 



R{x,y) = a 



(39) 
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R(^,y) = ^^^i^ " (40) 



[00047] In the case a < 0 the wall is visible only on the left 
side of the image (x < 0), and the reverse. A horizontal plane 
has the equation Yy = Jb in the environment coordinate system. 
With equation (4) the camera coordinates can be transformed 

Y cos X + Z sin T = jb (41) 
[00048] Introduced into equation (38) : 



R{x,y) = \ b (42) 



or in greater detail 



= Y'^'^^' b (43) 

4^cosr-h(4/ —X —yjsmT 



[00049] In the case Jb > 0, the plane is visible only below the 
horizon. 



Reflection o£ Qmnicam images 

[00050] For omnicams with an effective projection point, the 
omnicam- image can be reconstructed from a pinhole camera 
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perspective. [1] The centered pinhole camera coordinates are 
provided by the projection equation 



X' = 



y = 



f ^ 
z 

z 



(44) 
(45) 



[00051] The relationship between the omnicam and the pinhole 
camera coordinates are provided by a compression function, which 
can be derived from equation {21 t) : 



(46) 



[00052] A graph of this con^ression function can be seen in 
Fig, 5. 

[00053] The omnicam image position (x,y) corresponds to the 
centered pinhole camera coordinates (x',y') according to 



X = K 



/ 



(47) 



y = K 



f 



\ 



(48) 
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[00054] 



The inverse transformation is 



x' 



4/^ 



X 



(49) 



4f' ^x' 



y 



(50) 



4/' -x' -y' 



[00055] A graph of this coefficient 4/(4 - p^) is shown in Fig. 
6. 

Horizon and FOE 

[00056] For describing the effect of the vehicle movement in 
the image one uses the term horizon and expansion point (or FOE 
for focus of expansion) . Fig. 3 shows how a tilted camera is 
moved along by the vehicle, which moves along a horizontal 
plane. The horizon is the image of a plane parallel thereto, 
which passes through the projection center, as shown in Fig. 7. 
The main point of the camera is marked with "x", the FOE with a 
"+", the horizon has a yellow line. 

[00057] A straight line in the horizontal plane is provided by 
a point psj = (0,0,1)^ and the direction /v= (1,0,0)*^. A normal 
or perpendicular in the horizontal plane is nv = (0,0,1)^, and 
has the camera coordinates n = (0, cos t, sin x)^. As discussed 
above, one must distinguish between the two cases. 

• for camera tilt T = 0, the horizon is a straight line 



with y = 0. 
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• for X # 0, the horizon is a circular arc of which the 
center point and radius are defined by 

Xh = 0 (51) 
Yh = 2f cot T (52) 
rH = 2f / sin x (53) 

[00058] For a straight linear horizontal movement shown in 
Fig. 3 the FOE lies in the center of the horizon: 

Xfoe =0 (54) 

[00059] The ordinate is determined for x # 0 

ypoA = yn - ^ -2/ tan ^ (55) 



as follows also from Formula (26) . 



[00060] In a symbolic representation of the free space by a 
grid pattern it is precisely the grid elements which are 
comprised of perpendicular and horizontal segments of straight 
lines Pi, li, according to the discussion regarding Formula (29) 
through (34), wherein the pi represents the model point-of-space 
points on the straight line segments, and li respectively 
represent one of two unity vectors in the direction of the 
corresponding grid elements. These straight line segments are, 
as discussed, transformed into circular arc segments on the 
image sensor surface and these are represented as grid patterns. 
The locationally fixed representation of the symbolic grid 
elements are then accomplished thereby, that in the case of a 
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t • 

movement along the segment s in the longitudinal direction of 
the vehicle - measured for example by a rotary encoder or a 
speed sensor at the wheel - all model points or space points li 
are displaced or moved along the segment s opposite to the 
direction of travel . 



wherein e represents a unity vector in the vehicle longitudinal 
direction in the camera coordinate system. 

[00061] It is further useful, that the driver need not 
continuously move his vehicle in order to determine a danger of 
objects in the described manner on the basis of the movement 
pattern in the image. It should be made possible to him, for 
example by operating a switch, to generate a repetition of the 
last movement sequence on the display system. This can for 
example be the image material of the last five-meter travel 
segment with ''driving tube" included, shown in an endless loop. 

[00062] Possible design or presentation elements of the 
symbolic driving tube, which satisfy the requirements for the 
clear geometric structure and the transparency described in the 
last section are for example the representation of the edge 
boundaries by means of a grid pattern, a sequence of columns or 
a checkerboard pattern. 

[00063] Fig. 1 shows an image scene 10, in which the edge 
boundaries of the free space are visualized by superimposing 
symbolic represented walls 16. The image data associated with 
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the rear scene are distorted with the advantageous employment of 
a catadiopoteric camera. For a vehicle operator however, the 
essential object in the environment, such as for example the 
vehicle 11 and 12 as well as the garage 13 with the roll-up door 
14 is very easy to recognize and understand. In the shown 
example there is formed, by the use of the catadiopoteric 
camera, the part area 15 lying in the direction of travel of the 
own vehicle at the lower edge of the displayed or presented 
image data. Such a display has the advantage, that the vehicle 
operator is able to observe those areas of the vehicle having 
the greatest potential of collision danger. 

[00064] Without superimposing upon the image data the symbolic 
walls 16 it would be difficult for the vehicle operator however 
to estimate whether he can drive into the garage 13 with his 
vehicle, without bumping against the partially lowered roll-up 
door 14. The symbolic walls show him however that for his 
vehicle there is sufficient free space below the roll-up door, 
in order to drive into the garage without collision. From the 
straight forward progression of the symbolic walls, the vehicle 
operator can derive that his vehicle is optimally oriented for 
driving into the garage, and see that there is no longer any 
need for any essential steering correction during operating the 
vehicle. 

[00065] Fig. 2 shows an image scene (20) identical to Fig. 1, 
wherein here the edge boundaries of the free space are 
visualized by superimposing a symbolic represented virtual 
driving tube (26) with checkerboard pattern. By the use of a 
checkerboard pattern for accentuating the edge boundaries, the 
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observation of the image scene on the display becomes clear, and 
the movement of the own vehicle is particularly understandable. 
This is particularly advantageous in the situation where driving 
must occur at slow speed due to close proximity. 

[00066] By the use of clear geometric structures in the 
already discussed example of the symbolic driving tube, in 
combination with a registering lens, for example a 
catadiopoteric lens, a further advantage is achieved: by the 
reference structure of the symbolic driving tube displayed in 
the same manner, it becomes substantially easier for the driver 
to evaluate and take into consideration without having to think 
of the distortions of the lens. This is particularly 
conspicuous in the examples shown in Figs. 1 and 2, at the edge 
of the image to the right and left, in which the view onto a 
diagonal drive path is to be seen. For example, the warped 
boundaries of the rectangular driving tube shown in Fig. 2 
facilitates the interpretation, as to which of the elements 
lying above and below are in the scene. In the distortion shown 
here the street lies ''inside" as the lower element, and the sky 
lies at the outer edge of the circular- shaped representation. 

[00067] In particularly advantageous manner the invention is 
suited for rear-space monitoring during a re-orienting or 
repositioning operation of a motor vehicle. Herein the rearward 
space is often not directly observable by the vehicle operator, 
so that he is dependent upon a particularly clear depiction of 
the environment and the interaction of his vehicle therewith. 
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[00068] A further supporting and assisting of the driver 
during repositioning or re-orientation can be achieved, when the 
above described comparison of the movement of a locationally 
fixed driving tube with the actual movement of objects in the 
image are no longer left to him on the basis of how things 
appear, but rather are undertaken automatically by a computation 
unit. This is possible with the utilization of, for example, 
processes known in robotics for the comparison of the measured 
optical flow in the image with the calculated optical flow of a 
model vector field, here the model of a geometically defined 
driving tvibe corresponding to Fig. 13. 



[00069] The algorithm necessary for the automatic evaluation 
and comparison of the optical flow is described in the following 
by way of example for the use of a camera system comprised of a 
catadiopoteric camera (omnicam) with a parabolic mirror. Of 
course these basic principles can, in analogous manner, also be 
applied to other camera systems and other optical 
characteristics . 



Optical Flow 
General Optical Flow 

[00070] The derivative of the omnicam projection Equation 
(27f) is 
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[00071] If one again introduces the projection Equation {21 f) , 
there results 
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Optical Flow for Tilted Camera 

[00072] For a tilted camera as in Fig. 3 the movement is 
described by Equation (5f ) . If one introduces this into the 
Equation (58f) above, there results 
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[00073] Therein one must take into consideration that R and z 
depend upon the image position. In greater detail, the vertical 
flow says that 
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[00074] The distance i?(X»Y) modulates the horizontal and 
vertical flow with the same front or lead factor. Thus R{x>y) 
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influences only the length of the flow vectors, however not 
their direction. As shown, the direction of flow is determined 
only by the camera movement, that is, through the bearing or 
location of the point of expansion. For this translation 
movement the direction of the flow vectors is also independent 
of the depth structure of the environment. 



Optical Flow of the Camera in the Direction of Travel 

[00075] For an omnicam, of which the axis is oriented in the 

direction of travel, the flow Equation (60f) is simplified: 



Vx 

X = — (63) 
K 

Vy 
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[00076] This is very similar to the field of flow of the 
pinhole camera. The only difference is comprised in that the 
denominator R, in the case of the pinhole camera, would be Z. 
In both camera types the flow field is in this case purely 
radial, that is the flow vector (x,y) is parallel to the radius 
vector (x,y), also when i?(x,y) changes over the image. For a 
front parallel wall with a depth z = c there follows from 
Equation (38) 
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flow field appears as shown in 
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Optical Flow of a Vertical Side Wall 

[00078] One then takes into consideration again a camera with 
tilt angle T at the vehicle as in Fig. 3. The vertical side wall 
parallel to the direction of travel is determined by X = a. The 
distance of the wall according to Equation (39) can be utilized 
in the general flow Equation (60f ) : 
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[00079] 



In greater detail, this means 
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[00080] This is illustrated in Fig. 10 



Optical Flow of a Horizontal Plane 

[00081] A horizontal plane (for example a street or a garage 
roof) is defined by Yv = Jb. The wall distance according to 
Equation (42) can be introduced into the general flow Equation 
(60f ) : 
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[00082] In greater detail this thus means 
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[00083] This is illustrated in Fig. 11. 

[00084] For simplification, one can employ the third binomial 
series : 
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Optical Flow of a Corridor 

[00085] Fig. 12 shows the depth map of a corridor. The side 
columns show the colors employed for encoding depth. The depth 
is calculated according to Equation (39) for the vertical walls 
and according to Equation (42) for the horizontal floor and the 
ceiling. 

[00086] Fig. 13 shows the theoretical expected flow for this 
corridor. It can also be calculated according to Equation (7 Of) 
for the vertical walls and Equation (75f) for the floor and 
ceiling. Horizon and FOE are marked as above as the yellow 
circular arc or, as the case may be, a plus sign. Figs. 14 and 
15 show the same flow field as color encoded images respectively 
for the horizontal and vertical flow. 
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[00087] Those locations in the image, in which the model flow 
is less then the actual measured flow, are potential collision 
candidates and are displayed as such to the vehicle operator in 
a noticeable or standout color, and in the case of sufficient 
magnitude can also be employed as trigger for generating a 
warning signal, for example an acoustic warning signal. 
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